Papers with multi-choice multi-correct questions
“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility (2023.eacl-main)
Copied to clipboard
Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, Chitta Baral
| Challenge: | Recent work has found that large-scale language models lack commonsense reasoning ability . a dataset evaluating large-level language models is needed to evaluate their understanding of feasibility . |
| Approach: | They propose a question-answering dataset that tests understanding of feasibility . they propose to use commonsense reasoning to reason about when an action is feasible . |
| Outcome: | The proposed dataset shows that state-of-the-art models struggle to answer feasibility questions correctly. |